System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display

ABSTRACT

Video sources and inertial sensors are attached to a weapon and to goggles. A computer receives video images from the weapon- and goggles-mounted sources and inertial data from the sensors. The computer calculates a location for an image from the weapon-mounted source within an image from the goggles-mounted source using the inertial sensor data. The sensor-based location is checked (and possibly adjusted) based on a comparison of the images. A database contains information about real-world objects in a field of view of the goggles-mounted source, and is used to generate icons or other graphics concerning such objects.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/483,086, titled “System and Method for Video Image Registrationand/or Providing Supplemental Data in a Heads Up Display” and filed Apr.10, 2017, which application is a continuation of U.S. patent applicationSer. No. 14/950,643, titled “System and Method for Video ImageRegistration and/or Providing Supplemental Data in a Heads Up Display”and filed Nov. 24, 2015, which application is a continuation of U.S.patent application Ser. No. 11/680,207, titled “System and Method forVideo Image Registration and/or Providing Supplemental Data in a HeadsUp Display” and filed Feb. 28, 2007. U.S. patent application Ser. No.11/680,207, U.S. patent application Ser. No. 14/950,643, and U.S. patentapplication Ser. No. 15/483,086, in their entireties, are incorporatedby reference herein.

BACKGROUND

Augmented reality systems offer a mechanism for providing a user withadditional data about his or her environment. An example of such asystem is a heads-up display (HUD) found in aircraft and automobiles. Asis generally known, a HUD projects various navigational and/oroperational data on a windshield or other transparent surface in auser's field of view. This allows a user to monitor various informationwithout having to divert his or her gaze from the external environment.

Augmented reality systems have also been developed for use in a combatmilitary environment. For example, commonly-owned U.S. patentapplication Ser. No. 11/000,934 (filed Dec. 2, 2004, titled “System andMethod for Video Image Registration in a Heads Up Display,” published asPub. No. US20060121993, and incorporated by reference herein) describesa system in which an image from a rifle-mounted video source issuperimposed on an image seen through a pair of goggles. Sensors coupledto the rifle and to the goggles provide data indicating movement of thegoggles and rifle. An image from the rifle-mounted source shows anexternal region within the source's field of view (FOV). The goggleshave a wider FOV and provide an image that includes a portion showingthe same region as is shown in the image from the rifle-mounted videosource. The sensor data is then used to determine the relativeorientation of the two sources and calculate a location for the rifleimage within the image seen through the goggles.

Determining the relative orientations of two video sources based oninertial measurement unit (IMU) sensor data, as is described for atleast some embodiments of the aforementioned application, can posechallenges. For example, many low-cost IMU sensors experience bias driftover time. Such drift can result in relative orientation errors ofseveral degrees per hour. In order to prevent such errors fromaccumulating, the IMU sensors must be periodically recalibrated. Thisrecalibration typically requires user action and can disrupt systemoperation. The ability to minimize the need for manually-initiatedrecalibration would be highly advantageous.

Additional advantages could also be obtained by increasing the types ofinformation provided by a HUD within goggles worn by a user.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In at least some embodiments, a computer receives images from two videosources. Each of those two video sources is movable independent of theother and generates images that represent a portion of an externalenvironment within its field of view. One of the video sources may becontained within a pair of goggles worn by a user, and the other sourcemay be mounted to a rifle carried by the user. Sensors coupled to thetwo video sources provide data to the computer that indicates thespatial orientations of those sources. Using the sensor data, thecomputer determines a location for placing a video image (or a portionthereof) from a second of the sources (e.g., a rifle-mounted source) inthe video image from a first of the sources (e.g., a goggles-mountedsource). The second source video image and a corresponding portion ofthe first source video image represent the same part of the externalenvironment. Data from the two images are then compared in order toevaluate the location determined from the sensor data. The sensor-basedlocation is either confirmed, or a new location is found based onadditional image comparisons. Once a location is selected (either aconfirmed sensor-based location or a location found using imagecomparison), the two images are displayed such that the second sourceimage (or a portion of that image) overlays a corresponding portion ofthe first source image. Locations obtained using image comparisons areused to calibrate (adjust) the manner in which subsequent sensor-basedlocations are determined.

In at least some additional embodiments, a database contains informationabout external objects in the environment in which a system employingthe two video sources is being used. The computer receives systemlocation information and orientation information for one of the videosources (e.g., the source contained within a user's goggles). Thecomputer uses the location and orientation information to calculate aspatial volume within the field of view of that video source. Thecomputer then parses the database for information about objects that maybe located within the calculated spatial volume. After parsing thedatabase, icons or other graphical indicia are displayed for objectsidentified as being within the calculated spatial volume, the positionsof the icons being imposed upon the locations of the objects.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements.

FIG. 1 illustrates a system, according to at least some embodiments, forproviding an information-enhanced heads up display (HUD).

FIG. 2 is a block diagram of the control unit for the system of FIG. 1.

FIG. 3 is a block diagram showing data flows in the system of FIG. 1.

FIG. 4 is an example of a user display in the system of FIG. 1.

FIGS. 5A and 5B are a flow chart explaining operation of the system ofFIG. 1.

FIGS. 6A through 6F illustrate positioning of one video image withinanother based on data from inertial measurement units.

FIGS. 6G and 6H illustrate rotation of one video image within anotherbased on data from inertial measurement units.

FIG. 7 is a flow chart providing additional details for a block in theflow chart of FIGS. 5A and 5B.

FIGS. 8A through 8K illustrate checking and/or correcting an IMU-basedposition for one video image within another video image.

FIGS. 9A through 9C illustrate correction of an IMU-based positioncalculation based on image comparison results.

FIG. 10 is an example of a tactical database.

FIG. 11 is a flow chart providing additional details for another blockin the flow chart of FIGS. 5A and 5B.

FIGS. 12A through 12D show identification of real-world objects based ona spatial volume calculated from a position and orientation.

FIGS. 13A and 13B show a goggles images and scope image, respectively,used to describe an alternate image comparison algorithm.

FIG. 14 is a flow chart for an alternate image comparison algorithm.

FIG. 15 is a partially schematic diagram corresponding to the scopeimage of FIG. 13B.

FIG. 16 is a partially schematic diagram corresponding to the gogglesimage of FIG. 13A.

FIGS. 17A through 17C show collection of gray-scale values from scopeand goggles images.

FIG. 18 is a plot showing an example of gray-scale values.

FIG. 19 is a plot of the results of calculations in which image regioncenter points coincide.

FIG. 20 is a plot of the results of calculations similar to those usedto generate the plot of FIG. 19, but using random data.

FIGS. 21 and 22 show a pattern by which additional points are selected.

DETAILED DESCRIPTION

FIG. 1 illustrates a system 10, according to at least some embodiments,that provides an information-enhanced heads-up display (HUD) for aninfantryman or other armed tactical operator. For simplicity, a soldieror other person using system 10 will henceforth be referred to as a“user.” System 10 includes a set of goggles 11 that are configured forwear by the user. Goggles 11 may be worn directly, may be attached to ahelmet or other headgear (not shown), or worn in some other manner. Thefront side of goggles 11 faces outward (i.e., away from the user) whenworn, and includes eyepieces 12 and other apertures (not shown) forreceiving light or other (e.g., IR) input from the user's field of view.The rear side of goggles 11 faces the user when worn such that the userlooks outward through eyepieces 12 and sees external objects within thegoggles field of view. As explained in more detail below in connectionwith FIG. 3, goggles 11 includes an image generator and a projector andcreates a user display that combines video images and graphical indiciawith what a user sees as a result of light passing from the externalenvironment through eyepieces 12. A sensor 13 is attached to goggles 11or otherwise physically coupled so as to move as a single unit withgoggles 11. Sensor 13 includes an inertial measurement unit (IMU) andmagnetometer. In at least some embodiments, sensor 13 is an IM³™ sensor(available from Archangel Systems, Inc. of Auburn, Ala.) that usesaccelerometers and rate sensors to sense and report angular orientationaround three axes of rotation up to 100 times per second. The imageprojector and generator in goggles 11 and sensor 13 communicate (overcables 15 and 14, but by wireless means in at least some otherembodiments) with a wearable control unit 16. Control unit 16 includes acomputer, radio receiver and other elements, as is also described below.

System 10 further includes a video source (or “scope”) 17 and a sensor18 configured to move as a single unit with scope 17. In the embodimentshown, scope 17 is affixed to a rifle 19 and is used to target rifle 19.In particular, scope 17 is aligned with the barrel of rifle 19 such thata reticle corresponding to the optical centerline of scope 17 generallyfollows the path of a bullet fired from rifle 19. Scope 17 may include avisible-light video camera, a thermal imaging (i.e., IR sensitive) videocamera, a night-vision video camera, or some combination thereof. Incertain embodiments, scope 17 is a Light Weapon Thermal Sight (availablefrom DRS Optronics of Melbourne, Fla.). Sensor 18 is also an IMU (suchas the aforementioned IM³™ sensor) in at least some embodiments. Sensor18 and scope 17 communicate with control unit 16 via cables 20 and 21.

As is also explained in more detail in conjunction with FIG. 3, controlunit 16 receives video feeds from scope 17 and goggles 11. The videofeeds may be delivered using any standard video format, for exampleanalog formats like NTSC or PAL, digital formats like MPEG, or anon-standard format. Sensors 13 and 18 communicate angular pitch, yaw,and roll information to control unit 16. Using information from sensors13 and 18, a computer within control unit 16 determines orientation ofthe line of sight for goggles 11 relative to the line of sight for scope17. As described in more detail below, this permits superimposition ofan image from scope 17 onto a display corresponding to a field of viewof goggles 11.

In at least some embodiments, sensors 13 and 18 communicate viaUniversal Asynchronous Receiver/Transmitter (UART) protocol on cables 14and 21. These cables, along with video cables 15 and 20, may be sewninto a soldier's clothing, rifle sling, equipment harness, etc. toprevent entanglement. Although wired sensors and video cables are usedhere, control unit 16 communicates wirelessly with sensors 13 and 18,goggles 11 and/or scope 17 in other embodiments. For example,ultra-wideband (UWB) transceivers may transmit video and sensor datafrom rifle 19, as well as video and sensor data from goggles 11.Likewise, UWB may be used to transmit video from the control unit togoggles 11. Although UWB radios, such as Time Domain's PulsON® radio,are particularly desirable for their high bandwidth, low powerconsumption and for being virtually undetectable, any wireless standardmay be used, including both Bluetooth and IEEE 802.11.

In alternative embodiments, UWB radios may be used for more thantransmission of video and sensor data. Multiple radios may be placed onrifle 19 and on goggles 11 (or on a helmet, to which the goggles may beaffixed), each of which can relay their precise position. In thisfashion, control unit 16 is able to calculate the alignment of the rifleand goggles based on the relative location of radios rather than usingseparate orientation sensors.

FIG. 2 is a block diagram of control unit 16. Control unit 16 includes acomputer 30, global positioning system (GPS) chipset 31 and datacommunication chip set 32. GPS chip set 31 receives signals from GPSsatellites via antenna 33 and outputs GPS data to computer 30. Based onthat GPS data, computer 30 determines the location of system 10 (andthus of the user) to within the accuracy limits of the particular GPSchip set. For differential GPS systems intended for military use, thehorizontal position information is typically accurate to less than threemeters. RF data communication chip set 32 provides two-way datacommunication (using antenna 34) between system 10 and other elements ina tactical network. Those other elements may include additional systems10 (e.g., worn by other soldiers in a squad), a remotely locatedcommander, etc. The communication link(s) serviced by chip set 32 mayinclude a Land Warrior radio local net, single channel ground andairborne radio system (SINCGARS), Force XXI Battle CommandBrigade-and-Below (FBCB2) tracking system, etc.

Computer 30 includes a bus 35 connected to GPS chip set 31 and localcommunication chip set 32 via GPS interface 36 and communicationinterface 37, respectively. A processor 38 communicates with interfaces36 and 37 and with memory 40 via bus 35. Memory 40 includes acombination of volatile (e.g., random access memory, or RAM) andnon-volatile memory (e.g., removable and/or non-removable FLASH, EEPROM,hard drive, etc.). Stored in memory 40 are instructions for performingoperations of system 10 as described herein and a tactical database(described below). Additional components communicating with processor 38via bus 35 include a video input interface 41, video output interface 42and sensor interfaces 43-45. Video input interface 41 receives videoinput from scope 17 and goggles 11 and makes that input available toprocessor 38. Similarly, sensor interfaces 43 and 44 receive input fromsensors 13 and 18 and make sensor data available to processor 38. Othersensors (e.g., barometric sensors) may also provide input throughinterface 45. Processor 38 provides video output, via interface 42, togoggles 11 for presentation in a display generated within goggles 11.

FIG. 2 is merely one example of a control unit. In other embodiments,the components and/or functions shown may be combined and/or divided invarious ways. As but one example, a single input interface could receiveall sensor and video feed data. As but another example, some or all ofthe operations performed by processor 38 and other elements of controlunit 16 could be performed by one or more application specificintegrated circuits (ASICs), field-programmable gate arrays (FPGAs),etc. In some such embodiments, some or all of the operations describedherein as being performed by processor 38 are hardwired into one or moreASICs as gates and other logic dedicated to the calculations and otheroperations described herein.

FIG. 3 is a block diagram showing data flows in system 10. Scope 17receives electromagnetic input (e.g., visible light, infrared) fromwithin its field of view. The field of view for scope 17 is shownschematically in FIG. 3 with dotted lines and labeled “scope FOV.” In atleast some embodiments, the scope FOV is 18°. Also shown in FIG. 3 is anoptical centerline (or “line of sight”) for scope 17 (also referred toherein as the “scope FOV centerline”). For ease of explanation, it isassumed in certain of the following examples that the scope FOV is acone symmetrically centered on the scope optical centerline. This neednot be the case, however, and the invention includes embodiments inwhich the scope FOV is asymmetric and/or differently shaped and/or notcentered on an optical centerline. An example of an image from a scopewith a non-conical FOV is described below in connection with FIG. 13B.Scope 17 outputs a video feed corresponding to an image of externalobjects within the scope FOV. For simplicity, a video image output fromscope 17 will be referred to as a “scope image.” Sensor 18 is coupled toscope 17 (because of both components being attached to rifle 19) andoutputs data (P_(S), R_(S), Y_(S)) reflecting the rotational orientationof scope 17 about pitch (P), roll (R) and yaw (Y) axes (as shown in FIG.1). In a similar manner, sensor 13 coupled to goggles 11 outputs data(P_(G), R_(G), Y_(G)) reflecting the rotational orientation of goggles11 about pitch (P), roll (R) and yaw (Y) axes. Although one example ofpitch, roll and yaw axes is shown in FIG. 1, other axes may be used. Inat least some embodiments, sensors 13 and 18 output data atpredetermined sampling intervals (e.g. 100 Hz). As indicated above,sensor 13 also includes a magnetometer. The magnetometer senses theearth's magnetic field and provides a signal (MagNorth) indicative ofmagnetic north.

In addition to sensor data and a scope 17 video signal, control unit 16also receives periodic position data updates via antenna 33 and tacticaldata via antenna 34. As described above, position data via antenna 33includes data from GPS satellites. Tactical data includes data providingpositions and descriptions of various elements (e.g., friendly orhostile forces) in the environment in which system 10 is operating.Other types of data may be received from one or more additional sensors55. In at least some embodiments, sensor 55 is a barometer providingdata indicating changes in the elevation of system 10, which data can beused to confirm and/or adjust GPS-based elevation data.

Goggles 11 includes an array 56 of electromagnetic receptors (e.g.,charge-coupled devices, photodiodes or other photo-sensitive elements,microbolometers) that receive electromagnetic radiation (e.g., visiblelight, IR) from within the receptor array field of view (“goggles FOV”).As with the scope FOV, some of the examples herein assume that thegoggles FOV is a cone symmetrically centered about an opticalcenterline/line of sight (also referred to herein as the “goggles FOVcenterline”) extending from goggles 11. This configuration is notrequired, however. An example of an image from a goggles array with anon-conical FOV is described below in connection with FIG. 13A. Visiblelight from the goggles FOV also passes through eyepiece 12 and can beseen by the user. Although eyepiece 12 and array 56 are offset slightlyfrom one another, the eyepiece FOV (not indicated in FIG. 3) is the sameor larger than the array field of view (i.e., the goggles FOV in FIG. 3)and generally has the same center line. Array 56 generates image data(e.g., a night vision view of the environment within the goggles FOV).This image data is then provided to an image generator 57 that processesthe image data into an image of objects in the goggles FOV. Forconvenience, an image corresponding to the data from array 56 will bereferred to as a “goggles image.” As explained in more detail below, thegoggles image may be modified by computer 30 prior to being displayed toa user. Accordingly, the goggles image is also output via line 58 tocomputer 30. A video feed from computer 30 (which may contain thegoggles image modified to include the scope image and other graphics) isreceived by projector 59 via line 61. Projector 59, through beamsplitter 60, combines the goggles image with the visible light passingdirectly through eyepiece 12. The goggles image can also be provideddirectly to projector 59 from image generator 57 (e.g., if computer 30or scope 17 becomes inoperative but night vision capability is stilldesired). A heads-up display (HUD) field of view (HUD FOV) is also shownin FIG. 3, and is further discussed below.

For simplicity, FIG. 3 only shows one array, one eyepiece, one projectorand one beam splitter. In at least some embodiments, there is a separatearrangement of array, eyepiece, projector and beam splitter for eacheye. Goggles 11 in some embodiments is a modified version of theAN/PVS-21 low-profile night vision goggles with M-2755 Enhanced Heads UpDisplay available from Sensor Technology Systems, Inc. of Beavercreek,Ohio. The modifications to a standard AN/PVS-21 (or other night visiongoggles or enhanced vision system) required to implement variousembodiments of the invention will be apparent to persons of skill in theart once such persons are provided with the information included herein.

FIG. 4 shows an example of a user display 70 provided by goggles 11. Forconvenience, “user display” is used to refer to the image projected byprojector 59 (FIG. 3) through beam splitter 60 into eyepiece 12, andwhich is visible by a wearer of goggles 11. A user display may includethe goggles image (or a portion thereof), the scope image (or a portionthereof) and other graphical objects. The example of FIG. 4 assumessystem 10 is being used in a forest setting, though such a setting ismerely for purposes of illustration. Located within the goggles FOV (andthus in goggles image 82) are numerous trees and bushes such as might bepresent in a forest environment, as well as soldiers 71 (partiallybehind a tree in the lower left) and 72 (partially covered by foliage inthe upper right). User display 70 is also slightly shaded so as toindicate that the goggles image is a night vision display. The outercircle of FIG. 4 generally corresponds to the limit of the goggles FOV.The HUD portion of user display 70 is shown as a rectangular region 73in the center portion of the goggles FOV. As explained in further detailbelow, various graphical indicia are overlaid within HUD 73. Alsooverlaid on HUD 73 is a weapon view 74 corresponding to (and generatedfrom) the scope image. For convenience, some of the examples hereinassume that the weapon view is the same size as the scope image. This isnot a requirement, however. The scope image may be cropped to provide aweapon view that is actually smaller than the scope image (or thatcorresponds to reduced scope FOV). Although HUD 73 is shown as arectangle and weapon view 74 is shown as a circle in FIG. 4, othershapes could be used. In some embodiments, for example, weapon view 74is rectangular. Portions of the scope image at the outer edges of thescope FOV may similarly be cropped so as to give a weapon view a desiredshape.

As discussed below, the location and rotation of weapon view 74 withinuser display 70 is determined by computer 30 based on output fromsensors 13 and 18 and based on comparison of the scope image with thegoggles image. As rifle 19 is moved, scope images (or portions thereof)are dynamically positioned within user display 70 so as to indicatewhere scope 17 (and thus rifle 19) is pointing. In this manner, a userof system 10 is able to simultaneously survey a setting, acquire atarget and point a weapon at the target without having to remove his orher goggles and adjust to a weapon sight.

An aiming reticle 76 is located at the center of weapon view 74, andcorresponds to the rifle aim point (as zeroed to the rifle at aspecified range) and lies near the scope FOV centerline. Various othergraphical objects are also included in the HUD. For example, “friendly”icons 77 and 78 are positioned upon soldiers 71 and 72. In this manner,the user of system 10 is aware that soldiers 71 and 72 should not befired upon. Also shown in display 70 is an icon 79 indicating thelocation of (or direction to) a rally point (i.e., a position determinedin advance of a tactical operation and at which persons involved in theoperation may regroup). Icon 80 shows a reported location of an enemymachine gun position. Colored path 81 indicates a preplanned route.Additional details of generating the graphical objects in display 70, aswell as examples of other graphical objects that could be included indisplay 70, are provided below.

FIGS. 5A and 5B are a flow chart explaining the operation of system 10.The steps shown in FIGS. 5A and 5B (and in other flow charts describedbelow) can be reordered, combined, split, replaced, etc. For example,FIGS. 5A and 5B are not intended to indicate a requirement that allsteps be performed as part of a single algorithm. Many aspects of system10 operation could be accommodated on multiple independently-runningparallel program threads or as processes running in parallel on an ASIC,an FPGA, multiple processors, etc. Moreover, all of the operations shownin the flow charts of FIGS. 5A through 5B would not necessarily beperformed the same number of times as is suggested by the flow chart. Asbut one example, refreshing and/or updating of icons (or of certainicons) might occur at a different rate than updating and/orrepositioning of weapon view 74.

Beginning in block 101, a user activates system 10, and goggles 11,control unit 16 and scope 17 are energized. Sensors 13 and 18 are alsoenergized. The system is then calibrated (block 103). In at least someembodiments, a user of system 10 remains motionless for a period of timeso as to allow rate sensors and accelerometers in sensors 13 and 18 tostabilize, determine a down vector, etc. In at least some embodiments,IMU sensors 13 and 18 can both determine a common down vector based onthe earth's gravitational field. The sensors thus require no furtherinitial calibration as to the pitch and roll axes. However, each sensorrandomly chooses a zero-yaw orientation. Accordingly, sensors 13 and 18must be further calibrated so that a baseline difference between thezero-yaw orientation can be determined. For example, sensor 18 on rifle19 might arbitrarily choose a zero-yaw position that corresponds torifle 19 pointing along a 37 degree compass heading, and sensor 13 ongoggles 11 might arbitrarily choose a zero-yaw position that correspondsto the optical centerline of the goggles FOV pointing along a 246 degreecompass heading. In at least some embodiments, this difference inzero-yaw orientations is determined by the user looking at a distant(e.g., over 300 meters away) object through goggles. A calibrationreticle corresponding to the goggles FOV centerline is shown in thegoggles display. The user then moves rifle 19 so that a reticlecorresponding to the centerline of scope 17 is aligned with the gogglesreticle and presses a “calibration” button.

After initial calibration, computer 30 receives position data for system10 from GPS chipset 31 (FIG. 2) and/or from data communication chipset32 in block 105. In block 107, computer 30 receives data for a videoframe (i.e., a scope image) from scope 17 and data for a goggles imagefrom image generator 57. Operation then proceeds to block 109, wherecomputer 30 receives angular pitch (P_(G)), roll (R_(G)) and yaw (Y_(G))orientation data for goggles 11 from sensor 13. Computer 30 alsoreceives pitch (P_(S)), roll (R_(S)) and yaw (Y_(S)) data from sensor18. Data values from sensors 13 and 18 indicate the angle of verticalrise (pitch), the angle of horizontal rotation (yaw), and the angle ofrotation around the line of sight (roll), for both goggles 11 and scope17, which because of calibration share the same coordinate system.

In block 111, the pitch, roll and yaw angle values from (goggles) IMU 13and the pitch, roll and yaw angle values from (scope) IMU 18 are used tocreate an IMU-based rotation matrix. The IMU-based rotation matrix isthen used in block 113 to calculate a location (i.e., horizontal andvertical position) and an image rotation for the scope image within thegoggles image (and thus, the correct placement of weapon view 74 in userdisplay 70). Rotation matrices and the use of same to place and orientone image within another are known in the art and/or would be readilyapparent to persons of ordinary skill in the art once such persons areprovided with the information contained herein. As part of the IMU-basedlocation and image rotation calculation, computer 30 adjusts fordifferences between scope 17 (and a scope image) and array 56 (and agoggles image). For example, the goggles FOV in some embodiments is 40°and the scope FOV is 18°. Moreover, array 56 and a similar sensor arrayin scope 17 may have different numbers of pixels and/or different pixelpitches. Adjusting for these and other differences between scope 17 andgoggles 11 so as to determine a linear distance up or down on userdisplay 70 (and an image rotation) is within the ability of personsskilled in the art once such persons are supplied with informationprovided herein.

Turning briefly to FIGS. 6A through 6H, positioning and rotating aweapon view 74 within display 70 based on data from sensors 13 and 18 isgenerally illustrated. FIGS. 6A through 6H assume data from sensors 13and 18 is accurate (e.g., not significantly affected by bias drift) andthat weapon view 74 is generated based on the IMU sensor data only. Forconvenience, HUD 73 and various icons and other indicia included in FIG.4 are omitted from FIGS. 6B, 6D, 6F and 6H. In FIG. 6A, goggles 11 arelooking in exactly the same direction that rifle 19 is pointed. Thisresults in a centered weapon view 74 (FIG. 6B), and computer 30 placesan image from scope 17 directly over the center of goggles display 82.

In FIG. 6C, rifle 19 has yawed eight degrees to the left. FIG. 6Ddepicts the subsequent change in user display 70. The pitch and rollvalues detected by sensors 13 and 18 remain unchanged, but the yaw valuedetected by sensor 18 will change by minus eight degrees (or positiveeight degrees, depending on the chosen convention). When this iscompared to the values detected by sensor 13 (which have not changed inthe present example), the proper location for the scope image will be −8degrees from the HUD centerline. Accordingly, the placement of weaponview 74 in the heads up display is shifted to the left as shown (thesame result would have been obtained if rifle 19 was held still andgoggles 11 yawed eight degrees to the right). In at least someembodiments, for scope/goggles movements that would otherwise moveweapon view 74 beyond the HUD width, weapon view 74 remains at the edgeof HUD 73. In such a case, however, computer 30 gives weapon view 74 adistinctive border to indicate to the user that rifle 19 is pointingoutside the visual field of goggles 11. Weapon view 74 remains visiblein such embodiments so as to provide the user a view of where the weaponis pointing (e.g., allowing a user to aim and shoot around a corner).

In FIG. 6E, rifle 19 has been pitched upward six degrees. The yaw androll values detected by sensors 13 and 18 remain unchanged, but thepitch values from sensor 13 and from sensor 18 differ by six degrees.This results in weapon view 74 being shifted upward within display 70 asshown in FIG. 6F.

For ease of explanation, parallax is ignored in the explanation of FIGS.6A through 6F. Because video sources within goggles 11 and scope 17 areseparated by some distance (e.g. 0.5 meters, typically mostly vertical),the devices will in fact be looking at slightly different points even ifthey are in perfect alignment. As a result, in processing a video framefrom scope 17, the location where the frame is placed may be slightlyoff, and a displayed frame of video will not be aligned as perfectly aspossible. However, this problem diminishes as the distance to a targetincreases. For example, a target at 10 meters with 0.5 meters betweenscope 17 and goggles 11 produces an error of about 2.9 degrees in theplacement of a weapon view. At 100 meters, with the same 0.5 metersbetween scope 17 and goggles 11, the error is only 0.29 degrees. Forlarger distances, image comparison position calculations (describedbelow) compensate for errors caused by parallax.

At shorter distances, parallax is in many applications a non-issue. Atleast some implementations of system 10 are predominantly intended foruse against targets at distances greater than 10 meters. Moreover, whentargeting a weapon using system 10, weapon view 74 will ultimately bethe source of assurance that the weapon is pointed at a proper target.Even if weapon view 74 is slightly misaligned with the surroundingportions of the goggles view, the soldier will be primarily concernedthat the weapon is pointed at the correct target, and weapon view 74will still accurately show the intended target. Additionally, if it isknown a priori that targets will be at a short range (e.g., during aroom-clearing operation), calibration can be performed using a target atthe presumed average range of real targets to be encountered (ratherthan at 300+ meters), thus minimizing the effects of parallax.

FIGS. 6G and 6H show rotation of weapon view 74 in display 70. Based onthe data received from sensors 13 and 18, computer 30 calculates anamount by which weapon view 74 should be rotated within display 70 so asto be properly registered with the remainder of display 70. Variousalgorithms for rotating an image by a desired amount are known in theart. In the example of FIGS. 6G and 6H the image rotation of weapon view74 is ten degrees.

Returning to FIG. 5A, flow proceeds from block 113 to block 117 of FIG.5B. In block 117, the IMU-based calculation for position and rotation ofweapon view 74 within display 70 is checked using an image-based method.This permits computer 30 to adjust the manner in which IMU-basedpositions and rotations are calculated, thereby correcting for biasdrift and helping to maintain proper registration of the scope imagewithin the goggles image. As used herein, “registration” refers topositioning of a scope image (or a portion of that scope image) within agoggles image so that the two images are properly aligned andpositioned, and one image coincides with the other.

Although IMU-based orientation determination is relatively simple from acomputational standpoint and is generally quite robust over shortperiods of time, IMUs can have certain disadvantages. For example, IMUshave bias drift rates and may be susceptible to significant overshoot ifmoved rapidly. To address these concerns, the relative orientation ofgoggles 11 and scope 17 can be independently deduced by processing imagedata from image generator 57 and scope 17 if there is sufficient imagecontent and contrast and if similar imaging technologies (e.g.,microbolometers, CCD, etc.) are used. By checking the IMU-based positionand image rotation calculations using an image comparison, the manner inwhich data from IMU sensors is used can be periodically recalibrated andthe effects of bias drift, etc. reduced.

If certain conditions are satisfied, the image from scope 17 can beconsidered a dilated, translated and rotated distortion of the imagegenerated from image generator 57. The angle between the viewing vectorof array 56 (i.e., the goggles FOV centerline) and the viewing vector ofscope 17 (the scope FOV centerline) should not be too great (i.e., sothe image from scope 17 is contained within the goggles image). If theangle is too great as determined by IMU readings (such as when anoperator looks away from where the weapon is pointed), the imagecomparison algorithm is (in some embodiments) not invoked until the IMUdata indicates that the angle is within acceptable limits for imagecomparison. The distance between goggles 11 and scope 17 should also besmall compared to the distance from goggles 11 and scope 17 to objectsbeing viewed. This ensures that parallax effects do not disrupt asimplifying assumption that the goggles FOV centerline and the scope FOVcenterline are co-located.

In at least some embodiments, image-based position calculation relies onBrouwer's Fixed Point Theorem. This theorem holds that a continuousmapping from a planar region onto itself is guaranteed to have one pointwhose coordinates are left unchanged by the mapping. If the scope imageis a dilated, translated and rotated distortion of the goggles image,then there will be at least one point in the goggles image that has thesame position in both the goggles and scope images. For example, assumethe position of the fixed point is at (u,v) in local coordinates of thegoggles image and at (u′,v′) in local coordinates of the scope image.Brouwer's theorem holds that u=u′ and v=v′. The location of the fixedpoint constrains the solution of the proper orientation of the scopeimage relative to the goggles image up to a twist rotation. The twistrotation is about an axis corresponding to a line defined by the centerof projection of the scope camera and the point u′,v′ (i.e., the fixedpoint) on the scope camera focal plane. If the Brouwer fixed point forthe goggles and scope images is at the center of the scope image, thetwist rotation corresponds to the difference between the roll axisorientations of goggles 11 and scope 17.

The Brouwer fixed point can be found by scanning and comparing a gogglesimage and a scope image and determining a location where a rotationallyinvariant similarity metric is maximized. One such metric is the peak tosidelobe ratio (PSR) of a cross correlation of two linear signals. Oneof those signals is from a small annular region centered about alocation in a goggles image and the other is from an annular region (ofthe same angular size) centered about a location in the scope image. Ifthe PSR for those two annular regions is the maximum, the centers ofthose annular regions represent the Brouwer fixed point. Numerousdefinitions of PSR are known in the art and can be used in differentembodiments. For example, PSR can be defined as (PEAK−mean)/σ, wherePEAK is a peak cross correlation value for the centers of the twoannular regions, and where mean and σ are the respective mean andstandard deviations of cross correlation values for the annular regions.

Once the Brouwer fixed point is found, and assuming the scope image andgoggles image are within a sufficiently close wavelength range (e.g., ifthe scope 17 and array 56 are both sensitive to IR at wavelengthsbetween 8 and 12 microns) the twist rotation about the fixed point canbe deduced from the phase shift of the two signals. The phase shift isproportional to the displacement of the cross correlation peak used inthe PSR calculation between the two signals. That is, starting from thezero point of each annulus (as defined by image orientation) and takingsignal values around the annulus, determine the angular displacementfrom zero for each peak. The difference between these two angles is thephase shift.

FIG. 7 is a flow chart providing additional details about the operationsperformed within block 117 of FIG. 5B in some embodiments. In block 202,computer 30 performs an image-based check on a point of interest in theportion of the goggles image identified (using IMU data from sensors 13and 18) as corresponding to the center of the scope image. “Point ofinterest” refers to a point in an image (a center of a portion of thegoggles image in the current example) that is being evaluated aspossibly corresponding to a point on the other image (the center of thescope image in the current example). If the image-based check passes,the values for the IMU-based rotation matrix (and thus, the center forweapon view 74 predicted by that IMU-based rotation matrix) areverified, and computer 30 proceeds to block 204 on the “yes” branch. Ifthe image-based check fails, computer 30 proceeds on the “no” branch toblock 206. As described in more detail below, computer 30 will in block206 and in subsequent blocks expand the search (primarily in the globalyaw plane) to perform image-based checks on additional points ofinterest in the goggles image.

Turning briefly to FIGS. 8A through 8G, the performance of animage-based check according to at least some embodiments is explained.FIGS. 8A through 8G conceptually illustrate a case in which theimage-based check verifies the position for the scope image calculatedwith IMU data from sensors 13 and 18. As in FIGS. 6B, 6D, 6F and 6H, HUD73 and other elements shown in FIG. 4 have been omitted from FIGS. 8Athrough 8G. FIG. 8A shows a goggles image 82, and FIG. 8B shows a scopeimage 302. In the example of FIGS. 8A and 8B, the user is holding hishead level but has tilted rifle 19 twenty degrees (20°) to the left.Shown in FIG. 8A (with broken lines) is the portion 303 of goggles image82 where computer 30 intends (relying on the IMU-based rotation matrix)to place scope image 302. The center 304 of portion 303 is alsoindicated; so as to avoid confusion with the aiming reticle of theweapon view, the center is shown as a “+” inside of a circle. The brokenline 303 in FIG. 8A only shows the portion of goggles image 82 overwhich scope image 302 of FIG. 8B would be overlaid. The intendedrotation of scope image 302 within goggles image 82 (according to theIMU-based rotation matrix) is shown in FIG. 8A as the arcuate dimension“20°.”

In block 202 of FIG. 7, computer 30 performs an image-based check ofpoint 304 so as to evaluate the validity of point 304 as the center of aweapon view. Stated differently, computer 30 initially assumes thatpoints 304 and 305 are the Brouwer fixed point. As part of theimage-based check of point 304, several PSRs are calculated using aregion of goggles image portion 303 and regions of scope image 302. Asseen in FIG. 8C, a first PSR calculation is based on a cross correlationusing the pixel intensity values of the equally-sized annular regions308 and 309 respectively centered on the center points 304 and 305 ofgoggles image portion 303 and scope image 302. The PSR value from thiscalculation (PSR[304:305]) is then compared to PSRs calculated forannular regions centered on locations around center 305 of scope image302. For example, and as shown in FIG. 8D, another PSR (PSR[304:305-1])is calculated based on a cross correlation using the pixel intensityvalues of the equally-sized annular regions 308 and 309-1 respectivelycentered on points 304 and 305-1 of goggles image portion 303 and scopeimage 302. Similarly, PSR[304:305-2], PSR[304:305-3] and PSR[304:305-4]are calculated based on cross correlations using the pixel intensityvalues of equally-sized annular regions 308 and 309-2 (centered on point305-2, as shown in FIG. 8E), equally-sized annular regions 308 and 309-3(centered on point 305-3, as shown in FIG. 8F) and equally-sized annularregions 308 and 309-4 (centered on point 305-4, as shown in FIG. 8G).

Point 304 passes the image-based check (i.e., the assumption that points304 and 305 are the Brouwer fixed point is validated) when computer 30determines that PSR[304:305], PSR[304:305-1], PSR[304:305-2],PSR[304:305-3] and PSR[304:305-4] are within an acceptable range of oneanother. An acceptable PSR range will depend on the characteristicperformance of the sensor technology (microbolometer, etc.) being usedin the imaging sources (scope 17 and array 56 in the present example),as well as on peculiarities of a given manufacturer's implementation ofa particular class of sensor. The acceptable PSR range for a givencombination of a goggles sensor type and a scope sensor type can bedetermined by comparing a series of exemplar images obtained usingsamples of those two sensor types. Determining an acceptable range iswithin the routine ability of a person of ordinary skill in the art oncesuch a person is provided with the information contained herein.

Although five PSR values were obtained in the image-based check of thepreceding example, this need not be the case. In other embodiments, animage-based check of a point of interest calculates PSR values usingregions centered on more or fewer points around the center 305 of scopeimage 302. Those points around center 305 could also have locations(relative to center 305) other than as shown for points 305-1 through305-4. In still other embodiments, PSRs for an image-based check of agoggles image point of interest are based on annular regions centered onpoints around that goggles image point of interest instead of annularregions centered on points surrounding the scope image.

Returning to FIG. 7, and as mentioned above, computer 30 moves to block204 from the “yes” branch of block 202 if the image-based check for theIMU-based center of the weapon view passes. In block 204, computer 30determines if there is a roll difference between the two images. Inparticular, computer 30 calculates the phase shift of the scope imageand goggles image and deduces a twist rotation from the phase shift. Ifthe difference between the twist rotation and the IMU-based rolldifference is within acceptable limits (e.g., less than five degrees),the IMU-based value (and thus, the rotation of the scope image withinthe goggles image predicted by IMU data) is verified, and operationproceeds to block 119 of FIG. 5B. If the difference between the twistrotation and IMU-based rotation is not within acceptable limits,computer 30 proceeds on the “no” branch from block 204 to block 218.Block 218 is discussed below.

Returning to block 202 of FIG. 7, a failed image-based check in block202 indicates that the IMU-based location for scope image 302 withingoggles image 82 may be incorrect. In other words, the scope image andthe part of the goggles image over which computer 30 intends (based onIMU data) to overlay the scope image do not sufficiently represent thesame part of the external environment. In such a case, computer 30proceeds on the “no” branch from block 202 to block 206. In block 206,computer 30 attempts to find the proper location for scope image center305 within goggles image 82 by comparing scope image 302 with differentparts of goggles image 82. In particular, computer 30 performsimage-based checks for points of interest in goggles image 82 that areslightly offset from a center calculated from IMU data. This isillustrated conceptually in the example of FIGS. 8H through 8K, whichassumes the same goggles image 82 and scope image 302 of FIGS. 8A and8B, but in which the IMU-based position and image rotation are notcorrect. FIG. 8H shows goggles image 82 and a portion 311 of gogglesimage 82 where computer 30 now intends (relying on inaccurate IMU-basedinformation) to place scope image 302. The center 313 of portion 311 andthe IMU-based image rotation angle (17°) are also shown. Forconvenience, the correct position of scope image 302 within gogglesimage 82 is shown with a different type of broken line.

FIG. 8I shows an enlarged area of goggles image 82 around center 313. Inblock 206, computer 30 performs image-based checks for each of points315 a through 315 h. As seen in FIG. 8I, these points are slightlyoffset from center 313. For example, the image-based check of point 315g is based on cross correlations of the pixel intensity values forannular region 316 (centered on point 315 g, as seen in FIG. 8J) andeach of equally-sized annular regions 309 through 309-4 shown in FIGS.8C through 8G. Similar image-based checks are performed for the points315 a-315 f and 315 h.

Returning to FIG. 7, computer 30 then proceeds to block 208. In block208, computer 30 determines if any of the image-based checks of block206 passed. If so, computer 30 proceeds to block 216, which block isexplained below. If not, computer 30 proceeds on the “no” branch toblock 210. In block 210, computer 30 expands its search and performsimage-based checks on points in the goggles image that are even furtheroffset from IMU-based center 313. This is shown in FIG. 8K, wherefurther offset locations 315 i through 315 hh are indicated. In at leastsome embodiments, the expanded search is biased in the global yaw plane.As shown in FIG. 8K, this results in a greater number of expanded searchpoints of interest in the horizontal direction than in the verticaldirection. The search is biased in the global yaw plane because biasdrift in the pitch and roll planes is an order of magnitude less than inthe yaw plane due to the steadying influence of a constant one-ggravitational signal in the accelerometers. Although not shown, thearrangement of points of interest checked during block 206 could also bebiased in the global yaw plane.

From block 210 (FIG. 7), computer 30 proceeds to block 212 anddetermines if any of the image-based checks of block 210 passed. If so,computer 30 selects the location (e.g., one of points 315 i through 315hh) for which the highest PSR was obtained (e.g., the highest PSR for apoint of interest that passed) and proceeds to block 216 along the “yes”branch. If none of the image-based checks in block 210 resulted in apass, computer 30 proceeds on the “no” branch to block 214. In block214, computer 30 determines if there are additional regions of gogglesimage 82 with which a portion of scope image 302 can be compared. If so,computer 30 returns on the “yes” branch to block 210 and further expandsits search to include locations that are farther away from center 313.If computer 30 determines in block 214 that there are no more regions ofgoggles image 82 with which scope image 302 can be compared, computer 30proceeds on the “no” branch to block 119 (FIG. 5B). In at least someembodiments, computer 30 treats a failure of all points of interest topass an image-based check as evidence that there is temporarilyinsufficient contrast and/or content in the scope or goggles image toperform a PSR calculation, or that the scope FOV is not within thegoggles FOV. Under such circumstances, any adjustment of IMU-basedposition calculation using image comparison results will wait until abetter set of scope and goggles images are available. Alternatively,calibration can be performed manually (typically in less than tenseconds). Adjustment of IMU-based position calculation is described inmore detail below.

When computer 30 does determine that a point of interest in the gogglesimage has passed an image-based check, computer 30 proceeds to block 216of FIG. 7 (either from the “yes” branch of block 208 or from the “yes”branch of block 212). In block 216, computer 30 identifies a portion ofthe goggles image centered on the point for which the highest PSR wasfound. Computer 30 then proceeds to block 218 and calculates the twistrotation of scope image 302. The calculations of blocks 206 through 216determine the amount by which the scope image is translated relative tothe goggles image. The relative rotation of the two images (if any) iscalculated separately in block 218. As indicated above, this calculationis in some embodiments based on the phase shift of scope image 302 andthe portion of goggles image 82 identified in block 216.

Computer 30 then proceeds to block 219, where the IMU-based gogglesportion (311 in the example of FIG. 8H) and rotation matrix are replacedwith the portion identified in block 216 and with the twist rotation(from block 218). If block 219 is reached from the “no” branch of block204, only the (IMU-based) roll is replaced. Computer 30 also adjusts themanner of IMU-based position calculation. This is illustratedconceptually in FIGS. 9A through 9C. FIG. 9A shows a goggles image 350,the center point 351 for an IMU-based location for a scope image and thecenter point 352 for an image-comparison-based location for the scopeimage. The value H_(|IMU) is the horizontal location of the scope imagein the goggles image calculated with data from IMU sensors 13 and 18.The value H_(|Image) is the horizontal location of the scope image inthe goggles image calculated using the image comparison techniquedescribed above. In FIG. 9B, value V_(|IMU) is the vertical location ofthe scope image in the goggles image calculated with data from IMUsensors 13 and 18; V_(|Image) is the vertical location of the scopeimage in the goggles image calculated using the image comparisontechnique described above. Although H_(|Image), V_(|Image) and V_(|IMU)are shown as linear distances in FIGS. 9A and 9B, persons ordinarilyskilled in the art will appreciate that angular differences between theorientations of goggles 11 and scope 17 directly correspond to (and canbe used to readily calculate) a rotation matrix that can be used tocorrect a calibration matrix that has drifted.

As shown in FIGS. 9A and 9B, the difference between H_(|IMU) andH_(|Image) (ΔH) and the difference between V_(|IMU) and V_(|IMAGE) (ΔV)are used to update the rotation correction matrix derived from theinitial calibration and subsequent updates until a new correction isavailable (e.g., as a result of a subsequent pass through blocks 202-219of FIG. 7). Finally, FIG. 9C shows values R_(|IMU) (representing thedifference between the roll orientations of goggles 11 and scope 17calculated with data from IMU sensors 13 and 18) and R_(|Image) (thetwist rotation calculated using the image comparison technique describedabove). R_(|IMU) and R_(|Image) are used (in a manner readilyappreciable by persons of ordinary skill in the art) to correctIMU-based roll calculations until a new correction is available. Ingeneral, corrections to pitch and roll are much smaller than those toyaw due to the presence of three-axis accelerometers in both IMUs thatyield a consistent gravity vector, thus correcting much of the drift inthese directions. In some embodiments, pitch and roll corrections fromimage registration will not be applied until they exceed apre-determined threshold because this effect due to gravity usually willkeep these axes in control.

From block 219 of FIG. 7, computer 30 proceeds to block 119 of FIG. 5B.In block 119, the scope image is cropped, resized and/or rotated. Theresulting image is then overlaid on the goggles image as the weapon view(block 121). A reticle (which corresponds to the scope FOV centerline)may be generated, but in some embodiments an integral scope imagereticle will be displayed as part of the image. As mentioned above, theweapon view may be given a desired shape (e.g., rectangular or circular)and angular dimension(s), and portions of the scope view outside of thedesired shape may be cropped.

Computer 30 proceeds from block 121 to block 123, where a tacticalobject database is updated. As discussed in connection with FIG. 4, andas described in more detail below, user display 70 graphically indicateslocations of various real-world objects of interest to the user. Datafor such objects is stored in a database, accessible by computer 30,located in memory 40 (see FIG. 2), and periodically updated by messagesreceived from other friendly units by means of the communications chipset 32. FIG. 10 shows an example of a database 401, according to atleast some embodiments, storing information about real-world objects. Inthe present example, the real-world objects are items of tacticalinterest to a soldier; accordingly, such items may also be referred toherein as “tactical objects.” FIG. 10 shows database 401 in tabular formfor purposes of explanation. Database 401 need not be stored ororganized as a table, however, and need not have other features shown inFIG. 10. A database such as database 401 could be stored in numerousother formats, could include information other than that shown in FIG.10, and/or could be stored as multiple separate databases.

The first column in FIG. 10 contains an identifier for an object ofinterest. For convenience, relatively simple identifiers are shown inFIG. 10. The identifiers could take any form, however. In some cases, anidentifier may be a name (e.g., of another soldier) or other charactersthat a human user would recognize, and that might be displayed in userdisplay 70 under some circumstances. The second column of FIG. 10indicates an object type. Although only four object types are shown indatabase 401, numerous other types of objects could be included. Forexample, mission objectives, friendly outposts or emplacements,obstacles, contaminated areas, magnetic headings, stadia marks and otheritems can be included within database 401 and shown in a user displaywith an icon or other indicator. The third column of database 401 giveslocation information for each object. The location information providesthe three-dimensional location of an object within a particular tacticalregion. In some cases, the tactical region may be relatively small(e.g., several square miles that a military unit will patrol). In othercases, the tactical region may be much larger. For simplicity, thelocation data in FIG. 10 is represented generically as <x coord>, coord>(with a lower-case “y” in this case referring to a linear distance axisand not yaw) and <z coord>. However, actual location information may bein the form geographical coordinates, military grid reference systemcoordinates, or in any other desired format. The fourth column of FIG.10 shows the type of icon or other graphical object used to mark orotherwise indicate a tactical object in user display 70. Although theicon type is shown graphically in FIG. 10 for convenience, an actualdatabase may contain a pointer or other data object corresponding to aparticular icon.

Although not shown in FIG. 10, terrain and other geographic informationis also stored in database 401. This geographic information includeselevations of points in the tactical region, true north corrections formagnetic north values obtained from a magnetometer in sensor 13,magnetic dip angle to determine if there is interference to the compass,etc.

Data can be input into database 401 in various ways. Some data might bedownloaded into database 401 (e.g., via an RS232 or USB port on computer30) from a laptop computer as part of planning for a mission. Examplesof such data could include planned routes, known enemy positions, etc.Other data is added to database 401, while system 10 is in use, based onperiodic updates over the radio communications link. Block 123 of FIG.5B represents such updating of database 401 via radio communications.For example, changing locations of friendly personnel can be relayedfrom other systems 10 used by such personnel. As another example, anenemy unit location may be reported by a soldier using system 10, andthat enemy unit location is then broadcast to other soldiers (each ofwhom is also equipped with a system 10). Radio updates may also comefrom sources other than (or in addition to) soldiers equipped withsystems 10. As but one example, a remotely located commander or anairborne observation unit may provide tactical information about enemyunits, changed mission objectives, etc.

After updating database 401, computer 30 proceeds to block 125 of FIG.5B. In block 125, computer 30 identifies tactical objects for which anicon or other graphical indicator is to be shown on user display 70.Computer 30 then generates a display icon (or other visual indicator)for each of the identified objects. FIG. 11 is a flow chart providingadditional details about the operations performed within block 125 ofFIG. 5B. FIGS. 12A through 12D are used to conceptually explain variousoperations of FIG. 11. Computer 30 would not necessarily generate thediagrams shown in FIGS. 12A through 12D as part of the operationsdescribed in connection with FIG. 11.

In block 501 of FIG. 11, computer 30 first determines the location ofsystem 10 based on GPS coordinates and/or other data received in block105 of FIG. 5A. This is shown in FIG. 12A, a plan view of a portion ofthe tactical region for which information is stored in database 401.Various friendly soldiers, enemy positions and other tactical objectsare shown using the same types of icons shown in FIGS. 4 and 10. Each ofthe tactical objects has one or more entries in database 401. Theposition of system 10 (and thus of goggles 11) is also indicated. Inblock 503 (FIG. 11), computer 30 then uses magnetometer output fromsensor 13, or (in case of magnetic interference) the yaw Euler anglecalculated from the IMU movements subsequent to the last acceptablemagnetic fix, to determine the direction of the goggles FOV (i.e., thedirection in which the user is looking). In particular, the value ofMagNorth (see FIG. 3) is corrected (using geographically-specificmagnetic deviation and true-north correction data in database 401) so asto determine the direction in which goggles 11 are pointed. This isshown in FIG. 12B. Also shown in FIG. 12B is the goggles FOV and HUDFOV. Flow then proceeds to block 505 of FIG. 11, where pitch axis datafrom goggles sensor 13 is used to determine the degree to which the useris looking up or down. This is shown in FIG. 12C.

Computer 30 then proceeds to block 507. After determining the directionin which goggles 11 are pointed, computer 30 calculates a spatial volumecorresponding to the boundary of HUD 73 in user display 70. This HUDvolume is a pyramid centered on the goggles FOV centerline and extendingoutward. As seen in FIGS. 2, 4 and 12B, the HUD FOV (e.g., 32°) isnarrower than and centered within the goggles FOV (e.g.,) 40°. The HUDpyramid is bounded by a plane located at a range (d) from goggles 11.This is shown in FIG. 12D. Computer 30 searches database 401 for allobjects within the just-calculated HUD pyramid. The range (d) is in someembodiments selected by the user. For example, a user might be moreconcerned about objects that are relatively close (e.g., within 500meters) than about those that are further away. Moreover, providingicons or other display indicia for distant objects may unnecessarilyclutter user display 70. In certain embodiments, range (d) may bechanged while system 10 is operating and the user display can bede-cluttered in other ways. As but one example, route information,control measures, etc. may be removed at the push of a button during afirefight so as to minimize potential distracters to the soldier, whilefriendly icons would be retained to help limit fratricide. Removedinformation could be restored by the operator when again needed.

In block 511 of FIG. 11, computer 30 then generates icons or otherindicia in user display 70 corresponding to the objects, identified inthe database search of block 509, within the HUD pyramid. In someembodiments, the size of the icons or other indicia is logarithmicallyscaled so that icons or other indicia for more distant objects aresmaller.

Computer 30 then returns to block 127 (FIG. 5B). If more video framesfrom goggles 11 and scope 17 are received (i.e., goggles 11 and scope 17remain ON), then computer 30 returns to block 105 of FIG. 5A. Otherwise,operation of system 10 terminates.

Additional Embodiments

Other embodiments include numerous variations and/or modifications ofthe above-described devices and operations. As but one example, theweapon view can be a magnified version of the scope view (either at alltimes or selectively at the request of the user). In such embodiments,the image-based comparison techniques described above can first beperformed using an unmagnified scope image. Once the proper location andorientation for the scope image are deduced, the scope image can bemagnified and cropped prior to overlay as the weapon view. Other typesof information (e.g., a battery life indicator, navigational headings,etc.) can also be selectively shown in a user display. Indeed, a usercan be provided with a great deal of flexibility in selecting the typesof information to be shown on the display. Such selection can take placeduring set-up or while the system is in use. For example, a soldier onpatrol may choose to see display indicators for routes, rally points,indirect fire targets, preplanned control measures, etc. When thesoldier is attacked or is about to enter a firelight, he or she mightthen place the weapon in a close-combat mode (e.g., by pressing a modecontrol button located on a weapon). In such a close-combat mode, theuser could avoid information overload by suppressing all informationexcept icons for friendly forces. In some cases, the soldier might evenchoose to suppress “friendly” icons and opt for an audible warning ifthe weapon is pointed at friendly personnel. As yet another variation, asoldier might configure his system to display icons for friendly forceswithin a first range (e.g., 300 meters) and for enemy positions within asecond range (e.g., 1000 meters), to display graphics for patrol routeswithin a third range (e.g., 500 meters), to plot control measures withroutes, etc.

As discussed above, location information can be provided via GPStransmissions. Location data can also be augmented by local informationsuch as timing and ranges from nearby soldiers also equipped with asystem such as described herein. Other types of navigational data (e.g.,WAAS, or wide area augmentation system) can also be used. Barometricpressure can be used to provide a relative elevation error and used as acheck against a GPS-based elevation, as well as during systeminitialization. A terrain database (e.g., within a database such asdatabase 401) can include values for magnetic deviation, gravitationalacceleration, dip angle, elevation (e.g., DTED, or digital terrainelevation data) and/or other quantities as a function of longitude andlatitude.

Data from various sources can be updated at various rates (e.g., videoframes at 30 Hz, IMU data at 100 Hz, GPS or other navigational updatesat 1 Hz, range data from other soldiers at 8 Hz). IMU accelerometers canhave bandwidths greater than 1000 Hz, resolution of 1.25 milligrams, and2% sensitivity. A magnetometer can provide outputs at 200 Hz and have asensitivity of 1 mGauss (1 part in 300-600 Earth field). Rate sensorscan have a 2% error and 100 Hz bandwidth, and barometric pressure dataprovided at 100 Hz and within 1-2 meters of resolution. Range data fromother soldiers may be within 1.3 meters of error. Of course, all ofthese values are only examples, and the invention is not limited tosystems having specific values for any of these quantities.

Other embodiments also include numerous variations on theabove-described methods for using a comparison of a scope image and agoggles image to check an IMU-based location and rotation for that scopeimage within that goggles image. For example, the operations describedin connection with FIG. 7 initially assume that the center for a scopeimage within a goggles image calculated from IMU data is correct. If theimage comparison suggests that the IMU-based calculation is not correct,a search of other possible center locations moves outward from theIMU-based center. Assuming an IMU drift rate of approximately 1°/hour, aprocedure such as that of FIG. 7 works well (and minimizes computationalload on a processor) if the scope and goggles point in the samedirection at least once every 10 to 15 minutes. However, the procedurecould be modified for IMUs having different drift rates or if it isexpected that the scope and goggles will not be looking in the samedirection for an extended period. As but one example, a system could beconfigured so that a user can manually initiate a search (e.g.,beginning at the goggles image center) across the entire goggles imagefor a scope image location. This could be used for initially calibratingthe IMUs, or as a re-calibration if the user has not pointed his weaponforward for an extended period (e.g., if the weapon was slung over ashoulder). The search patterns shown in FIGS. 8J and 8K are onlyexamples. Numerous other search patterns could be implemented.

In some embodiments, an alternate image comparison algorithm is usedinstead of (or in addition to) the operations described in connectionwith FIGS. 7-8K. FIGS. 13A and 13B show a goggles images and scopeimage, respectively, that will be used to describe this alternatealgorithm. Unlike the previous discussion, which assumed the fields ofview for both goggles array 56 and scope 17 were conical, the images ofFIGS. 13A and 13B instead assume that array 56 and a similar arrayinside of scope 17 are rectangular. In particular, FIG. 13A is a gogglesimage 601 generated when array 56 (FIG. 3) is a 320×240 pixelmicrobolometer array (8-12 micron range) having a 14.4° horizontal FOV,a 10.8° vertical FOV, and an 18° diagonal FOV. The data from array 56 isconverted into a 640×480 pixel NTSC format image for transmission tocontrol unit 16 and digitized for processing. FIG. 13B is a scope image603 generated when an array inside scope 17 (FIG. 3) is also 320×240pixel microbolometer array (8-12 micron range) having a 14.4° horizontalFOV, a 10.8° vertical FOV, and an 18° diagonal FOV. The data from thescope array is also converted into a 640×480 pixel NTSC format image fortransmission to control unit 16 and digitized for processing. In thepresent example, the horizontal, vertical and diagonal goggles fields ofview are the same as the horizontal, vertical and diagonal scope fieldsof view. However, the below-described alternative image comparisonalgorithm can also be used in connection with a goggles and/or scopeimage created from different arrays having different fields of view(including the conical fields of view described previously), as well asin circumstances when the fields of view for the goggles and the scopeare different in horizontal, vertical and/or diagonal planes. Adaptationof the below described alternative algorithm to such other arrayconfigurations is within the routine ability of persons of ordinaryskill in the art once such persons are provided with the informationcontained herein.

In both images 601 and 603, each pixel corresponds to approximately0.0225° of the FOV in each of the horizontal, vertical and diagonalplanes. As indicated above, however, each of images 601 and 603 is basedon data from a 320×240 pixel array. Each array pixel would thuscorrespond to approximately 0.045° of the FOV in each of the horizontal,vertical and diagonal planes. Moreover, gray-scale values from the320×240 pixel arrays are duplicated so as to create images 601 and 603.For example, a pixel in 320×240 array 56 outputting a gray-scale valueof G corresponds to a 2×2 pixel block in image 601, with all four pixelsin that 2×2 block also having a gray-scale value of G. The significanceof this will be apparent below.

Returning to FIG. 13B, a center 604 is also indicated. Surroundingcenter 604 is a circle 605 having a diameter corresponding to 4° of theFOV. The region within circle 604 will be overlaid on goggles image 601as a weapon view (similar to weapon view 74 of FIG. 4).

FIG. 14 is a flow chart for an alternate image comparison algorithmaccording to at least some embodiments. Various blocks in the FIG. 14flow chart will be further explained using diagrams and plots in FIGS.15-22. In some implementations, the alternative image comparisonalgorithm of FIG. 14 assumes that current IMU data is available fromsensors 13 and 18 (FIG. 3), that IMU drift since the last calibration(or image registration) has not been significant, and that IMU data isthus usable as a starting point in the algorithm (as described below).In at least some embodiments, random noise in IMU output rarely exceeds1 degree in yaw or ½ degree in pitch or roll. The FIG. 14 algorithm alsoassumes that center area 605 is wholly contained within goggles image601 (i.e., that the portion of the external environment corresponding tocenter area 605 is also represented by a region wholly contained ingoggles image 601) at time intervals such that the bias drift will beless than 2 degrees.

Similar to the algorithm previously described in connection with FIG. 7,the image comparison algorithm of FIG. 14 is entered from block 115 ofFIG. 5B. As with the algorithm of FIG. 7, the algorithm of FIG. 14 isalso performed by computer 30 (FIG. 2). In the first block of FIG. 14,computer 30 resets a counter (N). As described in more detail below, thecounter N is used to prevent excessive looping through the algorithm.From block 701, computer 30 proceeds to block 703 and locates a regionof high contrast, within scope image 603, that is relatively near center604. This is shown in more detail by FIG. 15.

FIG. 15 is a partially schematic diagram in which the various imageelements (e.g., trees, foliage, etc.) have been omitted for simplicity,but in which the boundary of scope image 603 and center 604 are shown.In block 703, computer 30 searches image 603 for a square region 610 inwhich the pixels inside of region 610 have a high degree of contrast. Inparticular, computer 30 searches for the square region 610 in which thestandard deviation of pixel gray-scale values is maximized. In thepresent example, and as indicated above, image 603 comprises numerous2×2 pixel blocks in which all pixels in a 2×2 block have the samegray-scale value. Accordingly, when calculating a standard deviation fora region of image 603, only every other pixel in the horizontal andvertical directions need be used. In at least some embodiments, region610 corresponds to a 3° horizontal FOV by 3° vertical FOV portion ofimage 603. In the present example of example of image 603, region 610 isapproximately 133 pixels by 133 pixels. In some such embodiments, thecenter 606 of region 610 should also be within 2° of center 604 in thehorizontal plane and within 1° of center 604 in the vertical plane. Inthe present example, computer 30 thus searches for a 133×133 pixelregion contained within a 311×222 pixel region of image 603 that iscentered about center 604.

Returning to FIG. 14, computer 30 proceeds from block 703 to block 705and determines if the region 610 identified in block 703 is ofsufficiently high contrast. If, for example, the maximum standarddeviation for gray-scale values within any 133×133 pixel block is notsufficiently high, there may be insufficient contrast to continue withan image comparison. A sufficiently high standard deviation value couldbe determined for a given combination of goggles and scope imaging arraytypes based on a series of test images. If the standard deviation forgray scale values in the region 610 identified in block 703 is notsufficiently high, computer 30 returns to block 119 of FIG. 5B via the“yes” branch. If the standard deviation of gray-scale values for theregion 610 identified in block 703 is sufficiently high, computercontinues with the image comparison and proceeds on the “no” branch toblock 707.

In block 707, computer 30 locates the point in goggles image 601 thatIMU data (from sensors 13 and 18) indicates is a representation of thesame part of external environment represented by point 606 (FIG. 15). Asindicated above, point 606 is the center of the region 610 found bycomputer 30 in block 703. The operation of block 707 is shown in moredetail in FIG. 16. FIG. 16 is a partially schematic diagram in which thevarious image elements (e.g., trees, foliage, etc.) have been omittedfor simplicity, but in which the boundary of goggles image 601 is shown.Point 607 is the point which IMU data indicates corresponds to point 606in the scope image. Also shown in FIG. 601 is a 1° by 1° (e.g., 44pixels by 44 pixels) region 613 centered on point 607. The significanceof region 613 is explained below.

Returning to FIG. 14, computer 30 proceeds from block 707 to block 709.In block 709, computer 30 obtains pixel gray-scale values from 128locations on a circle around center 606 of region 610 (FIG. 15). This isshown in more detail in FIG. 17A, a schematic diagram of pixelssurrounding center 606 in a portion of scope image 603. Each of thesquares in FIG. 17A represents one pixel of scope image 603. Surroundingcenter 606 are 128 locations (shown as diamonds) that are equally spacedon a circle having a diameter of approximately 1°. For each of thediamonds in FIG. 17A, computer 30 obtains the gray-scale value for thepixel in which the diamond's center is located. Unlike the calculationdiscussed in connection with block 703 (FIG. 14), however, values arenot limited to those for every other pixel. In other words, gray-scalevalues may be obtained for multiple pixels within a 2×2 pixel blockcorresponding to a single pixel of the microbolometer array in scope 17.Indeed, there might be instances in which two of the points on thecircle are located within the same pixel of image 603.

Returning to FIG. 14, computer 30 proceeds from block 709 to block 711.In block 711, computer 30 performs an operation similar to that of block709, but instead using goggles image 601. Specifically, computer 30obtains pixel gray-scale values from 128 locations on a circle aroundpoint 607 in goggles image 601 (FIG. 16). This is shown in more detailin FIG. 17B, a schematic diagram of pixels surrounding point 607 in aportion of goggles image 601. Similar to the calculation of block 709,gray-scale values may be obtained for multiple pixels within a 2×2 pixelblock corresponding to a single pixel of the microbolometer array 56 ingoggles 11.

In the present example, goggles image 601 and scope image 603 have thesame horizontal, vertical and diagonal fields of view and are of thesame pixel dimensions. This need not be the case, however. If one ofimages 601 or 603 was of a different pixel dimension or had a differentfield of view, the procedures of blocks 709 and 711 would be similar. Inparticular, gray-scale values would be obtained from pixelscorresponding to the same number of points distributed onsimilarly-dimensioned circles in each image. If, for example, a gogglesimage has a 32° horizontal FOV, a 24° vertical FOV and a 40° diagonalFOV, each pixel in that image represents approximately 0.1°. An evendistribution of 128 points on a 1° circle would be generally as shown inFIG. 17C. In FIG. 17C, point 607′ is a point (determined using IMU data)corresponding to the center of a high contrast region found in block 703(FIG. 14). Each square in FIG. 17C represents one pixel of a gogglesimage having the larger FOV values.

Returning to FIG. 14, computer 30 then proceeds to block 713. In block713, computer 30 performs various calculations using the gray-scalevalues obtained in blocks 709 and 711. FIG. 18 is a plot showing anexample of raw data that might be obtained in blocks 709 and 711. Theplots of FIG. 18 and of subsequent FIGS. 19 and 20 are provided forpurposes of explanation, and would not necessarily be generated bycomputer 30. In FIG. 18, the solid line plot represents the 128gray-scale values obtained in one of blocks 709 and 711, and the brokenline plot represents the 128 values obtained in the other of blocks 709and 711. For purposes of illustration, a 10% noise factor and a 22.5degree phase shift (representing a rotation of one image relative to theother) have been added. As part of the calculations of block 713 (FIG.14), computer 30 first performs a discrete Fast Fourier Transform (FFT)on each data set (i.e., a discrete FFT on the 128 scope image values anda discrete FFT on the 128 goggles image values). Computer 30 thenmultiplies the complex conjugate of one of those discrete FFTs by theother of the discrete FFTs. Computer 30 next obtains an inverse FFT ofthe product. Computer 30 then subtracts the DC component of the inverseFFT and obtains data that allows peak-to-sidelobe ratio (PSR)comparisons.

Computer 30 then evaluates the result of block 713 in block 715. If thecenter points of the scope image region and of the goggles image regionto which it was compared (e.g., points 606 and 607 of FIGS. 15 through17B) represent the same part of the external environment, the PSR willbe relatively large (e.g., 3 or higher). FIG. 19 is a plot of theresults of a calculation in block 713 (FIG. 14) when the center pointsdo coincide. The sidelobes in FIG. 19 can be seen as three regions (oneither side of the peak) that are higher than their surroundings. Thelocation of the peak can also be used to determine the rotation of oneimage relative to the other. In the example of FIG. 19, the peak isdisplaced 8 pixel increments from the left of the center of the plot.This corresponds to a rotation of (8/128)*360°, or 22.5°. The directionof rotation is determined based on whether the displacement is to theleft or to the right of the center. FIG. 20 is an example of a plotresulting from the calculations of block 713 using random data.

If the result of the evaluation in FIG. 14 block 715 is favorable (e.g.,if the PSR 3), computer 30 proceeds to block 717 via the “Fay.” branchand stores the PSR, rotation and the pixel coordinates of the point inthe goggles image 601 that was used in block 711 (e.g., point 607) asthe circle center. Computer 30 then continues to block 719. If theresult in block 715 is not favorable (e.g., PSR<3), computer 30 proceedson the “Unfav.” branch to block 719 without storing a PSR, centerlocation or rotation.

In block 719, computer 30 determines if there are additional points inregion 613 of goggles image 601 (FIG. 16) that should be checked aspossibly coinciding with point 606 of scope image 603. FIGS. 21 and 22show a pattern by which additional points in region 613 are selected inat least some embodiments. FIG. 21 schematically shows region 613 andlocation 607 (found in FIG. 14 block 707), with each of the squaresrepresenting a single pixel. FIG. 22 is an enlargement of the portion ofregion 613 indicated in FIG. 21. In the first pass through block 721after checking point 607, computer 30 selects the pixel at position{circle around (2)}. Computer 30 then returns to block 711 and repeatsblock 711 using pixel position {circle around (2)} instead of point 607,and then repeats blocks 713 through 719 (possibly omitting block 715).In the next pass through block 721, pixel position {circle around (3)}is selected. This continues until some stop condition is reached. Insome embodiments, a stop condition is reached when the pattern showngenerally in FIG. 22 reaches the sides of region 613 along thehorizontal plane. The pattern of FIG. 22 weights the search based on ahigher probability that any random noise or bias drift will be greateralong the yaw axis than along the pitch or roll axes. In otherembodiments, the stop condition is reached after every other point inboth the horizontal and vertical directions has been checked.

Once a stop condition is reached and there are no more points to selectwithin region 613, computer 30 proceeds to block 723 and determines ifany of the data sets (with each set containing a PSR, rotation andcenter point location) stored in block 717 is unambiguously the best. Inat least some embodiments, the “best” data set is one in which the PSRis a predetermined percentage above the other stored PSRs. Thepercentage by which a “best” point should exceed others depends on thecharacteristics of the imaging technology involved, but one examplevalue is an excess of 10%.

If an unambiguously best data set is identified in block 723, computer30 proceeds to block 725. In block 725 (which is substantially similarblock 219 of FIG. 7), computer 30 updates the IMU-based position forweapon view 74 based on the position and rotation of scope image region605 within goggles image 601 resulting from the unambiguously best dataset. Computer 30 also uses that data set to adjust the manner in whichsubsequent IMU-based position and image rotation calculations are made.This adjustment is performed as previously explained in connection withFIGS. 9A-9C. From block 725, computer 30 proceeds to block 119 of FIG.5B.

If an unambiguously best stored data set is not found in block 723,computer 30 proceeds to block 727. If counter N has reached a maximumvale (N_(MAX)), computer 30 returns (via the “yes” branch) to block 119of FIG. 5B. Appropriate values of N_(MAX) would depend of various designpreferences (e.g., processor speed, acceptable latency, etc.). Ifcounter N has not reached N_(MAX), computer 30 proceeds on the “no”branch to block 729 and increments counter N, and then proceeds to block731. In block 731 a new high contrast region is selected in the scopeimage. In some embodiments, the selection of block 731 identifies the3°×3° pixel block having the next highest gray-scale value standarddeviation, and which does not overlap a previously selected 3°×3° pixelblock in image 603 by more than a predetermined amount. Computer 30 thenreturns to block 707.

As with the algorithm described in connection with FIG. 7, numerousvariations on the algorithm of FIG. 14 are within the scope of theinvention. The search pattern of block 721, the sizes of image regionsevaluated, and various other parameters can be varied. As but oneexample, the above-described algorithm employs center 604 as onereference point and point 607 as a corresponding reference point. Inother embodiments, a point other than the center of the scope image canbe used as the initial reference point. As another example, individualpixel gray-scale values are the image element data points used in theFFT analyses. Other image element data points could be used (e.g., anaverage of gray-scale values for several pixels clustered around apoint, wavelength (color) information if sensors provide colorinformation). As yet another example, the diamonds in FIGS. 17A through17C have a particular dimensioned arrangement relative to the referencepoint (center 604) and the corresponding reference point (point 607). Inother embodiments, different arrangements (e.g., oval, square,concentric rings, etc.) and/or different dimensions (e.g., closer to orfarther from the reference point and corresponding reference point) canbe used. In still some other embodiments, data sets are stored in block717 for only the five highest (or some other predetermined number ofhighest) PSR values. In some embodiments, the scope image has a smallnumber of possible FOVs (e.g., 9° diagonal or 18° diagonal).

In some embodiments using the algorithm of FIG. 14, and in circumstanceswhere an image-based placement of the weapon view cannot be performedfor a significant period (e.g., if there is a long period in which thereare no sufficiently high contrast images), drift may become obvious tothe user. In such a case, the user can manually recalibrate the system.

Although embodiments of the invention have been described by example ofa system intended for use by a soldier or other combat personnel, theinvention is not limited in this regard. In other embodiments, a headsup display need not be associated with a pair of goggles. For example,the heads up display could appear before a windshield in a vehicle. Aweapon mounted on the vehicle includes a video gun sight producingimages processed and projected onto the heads up display. In thisembodiment, an orientation sensor may be placed to sense the orientationof the vehicle rather than a pair of goggles worn by the observer. Thisembodiment may be particularly useful for remotely controlled weaponsystems, for example a robot carrying a weapon.

It should also be noted that the techniques described herein are notlimited to weapon targeting or other combat uses. Other embodimentscould be used in a myriad of settings, including law enforcement,medicine, astronomy, etc.

Although examples of carrying out the invention have been described,those skilled in the art will appreciate that there are numerous othervariations, combinations and permutations of the above described devicesand techniques that fall within the spirit and scope of the invention asset forth in the appended claims. As used in the claims, “controller”generically refers to any of one or more FPGAs, microprocessors, ASICs,other types of computational devices, or combinations thereof. It is tobe understood that the subject matter defined in the appended claims isnot necessarily limited to the specific features or acts describedabove. Rather, the specific features and acts described above aredisclosed as example forms of implementing the claims. In the claims,various portions are prefaced with letter or number references forconvenience. However, use of such references does not imply a temporalrelationship not otherwise required by the language of the claims.

1. A video processing method for use with a system integrating a firstvideo camera configured to be mounted to a helmet and a second videocamera configured to be mounted to a weapon, the method comprising:receiving a first video signal from a first video camera; receiving asecond video signal from a second video camera; identifying, as afunction of information received in the second video signal, a region inan image from the first video camera; outputting, based on theidentified region in the image from the first video camera, a combinedimage reflecting a combination formed from information received from thefirst video camera and the second video camera.